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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 



- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 



3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [3 Claim(s) 1-69 is/are pending in the application. 

4a) Of the above claim (s) 11,12.23,24 and 35-37 is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) IEI Claim(s) 1-10, 13-22, 25-34, 38-69 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) K The specification is objected to by the Examiner. 

10)^ The drawing(s) filed on 10 May 2001 is/are: a)n accepted or b)KI objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .1 21 (d). 
11 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * 0)0 None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Status 



1 )^ Responsive to communication(s) filed on 10 June 2004 . 
2a)n This action is FINAL. 2b)^ This action is non-final. 



Attachment(s) 

1) S Notice of References Cited (PTO-892) 

2) mi Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) S Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 



4) □ Inten/iew Summary (PTO-413) 



5) □ Notice of Informal Patent Application (PTO-152) 

6) □ Other: . 



Paper No(s)/Mail Date. 



Paper No(s)/Mail Date 2/ 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 3 
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Detailed Action 



Election/Restrictions 

1. Restriction to one of the following inventions is required under 35 U.S.C. 121: 

I. Claims 1-10, 13-22, 25-34, 38-69, drawn to three dimensional modeling apparatuses, 
methods, and/or stored computer programs, classified in class 382, subclass 154. 

II. Claims 11-12, 23-24, and 35-37, respectively drawn to three dimensional shape data 
recording apparatuses, methods, and/or stored computer programs for recording three 
dimensional shape data of a three dimensional object, classified in class 382, subclass 
154. 

2. Inventions I and II are related as subcombinations disclosed as usable together in a single 
combination. The subcombinations are distinct from each other if they are shown to be separately usable. 
In the instant case, invention II has separate utihty such as recording 3D shape data, which may not, in and 
of itself, correspond to the data of a 3D model of an object (e.g. 3D temperature gradient of an object can 
represent 3D shape data), or may correspond to a 3D model generated by a modeling process, apparatus, 
and/or computer program other than invention I (e.g. CAD or CSG models). See MPEP § 806.05(d). 

3. Because these inventions are distinct for the reasons given above and have acquired a separate 
status in the art because of their recognized divergent subject matter, restriction for examination purposes 
as indicated is proper. 

Election By Telephone 

4. During a telephone conversation with Mr. Roger R. Wise on June 10, 2004 a provisional election 
was made with traverse to prosecute the invention of invention I, Claims 1-10, 13-22, 25-34, 38-69. 
Affirmation of this election must be made by applicant in replying to this Office action. Claims 1 1-12, 23- 
24, and 35-37 are withdrawn from further consideration by the examiner, 37 CFR 1. 142(b), as being drawn 
to a non-elected invention. 
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Drawings 



Objections 



5. 



The drawings are objected to because of the following. According to page 29, lines 15-22 of the 



Applicant's disclosure: 

As shown in FIG. 6, tables each corresponding to one of the variables X Y, Z, a, P, and y are prepared and all the values 



of X Y, Z, a, p, and y obtained for all the combinations are polled in the table. The value that forms a peak in each table 
is considered as the valid value. A point in FIG. 6 pointed by an arrow in the upward direction represents the valid value 
for that variable. 



Fig. 6 does not depict tables. Furthermore, no indication is given in Fig. 6 as to what the ordinate of the 
graphs depicted in Fig. 6 corresponds to. Illustration and labeling of an ordinate axis would clarify Fig. 6. 
See also the corresponding discussion below with regard to the Applicant's specification. 



. . .the light portion of pattern A (shown as (A) in FIG. 10) is (1 1 1 1) and the dark portion of pattern A is (0000), and the 
overall pattern can be represented by 8 bits, (111 10000). Similarly, pattern B (shown as (B) in FIG. 10) is (1 1001 100) and 



pattern C (shown as (C) in FIG. 10) is (10101010). 

This is clearly not depicted in Fig. 10. In Fig. 10, the dark portions of the depicted patterns correspond to 
codes of all I's, whereas the light portions correspond to codes of all O's. See also the corresponding 
discussion below with regard to the Applicant's specification. A proposed drawing correction or corrected 
drawings are required in reply to the Office action to avoid abandonment of the appUcation. The objection 
to the drawings will not be held in abeyance. 

7. Figures 9 and 15 should be designated by a legend such as ~ Prior Art — because only that which 
is old is illustrated. See MPEP § 608.02(g). Corrected drawing sheets are required in reply to the Office 
action to avoid abandonment of the application. The replacement sheet(s) should be labeled "Replacement 
Sheet" in the page header (as per 37 CFR 1.84(c)) so as not to obstruct any portion of the drawing figures. 
If the changes are not accepted by the examiner, the applicant will be notified and informed of any required 
corrective action in the next Office action. The objection to the drawings will not be held in abeyance. 



6. 



According to page 32, lines 20-25 of the Applicant's disclosure: 
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Specification 

Objections 

8. The disclosure is objected to because of the following informalities. 

a. The equation 1 referred to in page 28, lines 27 to page 29, lines 5 is not given. 

b. According to page 29, lines 15-22 of the Applicant's disclosure: 

As shown in FIG. 6, tables each corresponding to one of the variables X, Y, Z, a, P, and y are 

prepared and all the values of X, Y, Z, a, p, and y obtained for all the combinations are polled in 
the table. The value that forms a peak in each table is considered as the valid value. A point in FIG. 6 
pointed by an arrow in the upward direction represents the valid value for that variable. 

No tables are shown in Fig. 6. Instead, Fig. 6 depicts graphs, plots, or the like. Referring 
to these as such would be more descriptive of what is depicted in Fig. 6. Moreover, it is 
not clear from the specification or from Fig. 6 what exactly is being depicted in Fig. 6. 
Specifically, the graphs of Fig. 6 show individual relationships between the variables X, 
Y, Z, a, p, y and some dependant variable (the value of which extends along the ordinate 
direction). However, it is unclear from the Applicant's disclosure what that dependant 
variable is. 

Note that changes made to the specification to rectify these issues would obviate the need 
for the drawing corrections of Fig. 6 suggested above. 

c. On page 3 1, lines 26 of the Apphcant's disclosure, the Apphcant refers to a gypsum 
block. It is unclear from the Applicant's disclosiu*e as to what a gypsum block is. The 
Applicant's usage of the word gypsum\ is not consistent with anything found in the 
field of computer graphics and image processing, nor can its meaning, and hence the 
meaning of a gypsum block, be gleaned from the Applicant's specification. 

d. On page 32, line 7 of the Apphcant's disclosure, the Apphcant refers to "such a convex 
shape". The word convex should be replaced with the word concave in order to be 

1 Gypsum is used to make plaster. 



Application/Contn 
Art Unit: 2623 




imber: 09/847,011 



Page 5 




f. 
g- 

h. 
i. 

j. 
k. 



consistent with the rest of the disclosure (e.g. the inside of a cup - page 32, line 5 of the 
Applicant's disclosure - is concave not convex). Similar errors occur on page 36, hnes 
18, 22 and 26 and page 50, line 2 of the Applicant's disclosure. 
According to page 32, lines 20-25 of the Applicant's disclosure: 

. . .the light portion of pattern A (shown as (A) in FIG. 10) is (1 1 1 1) and the dark portion of pattern 
A is (0000), and the overall pattern can be represented by 8 bits, (111 10000). Similarly, pattern B 
(shown as (B) in FIG. 10) is (1 1001 100) and pattern C (shown as (C) in FIG. 10) is (10101010). 

This is clearly not depicted in Fig. 10. In Fig. 10, the dark portions of the depicted 
patterns correspond to codes of all 1 's, whereas the hght portions correspond to codes of 
all O's. Changing the specification to indicate this would rectify this matter and obviate 
the need for any of the proposed corrections to Fig. 10, discussed above. 
It is suggested that the word slashes on page 39, lines 8 of the Applicant's disclosure, be 
changed to cross-hatching or cross-hatched region. 

On page 39, lines 22 of the Apphcant's disclosure, (Date recording) should be changed to 
(Data Recording). 

On page 40, line 1 of the Apphcant's disclosure, the word Foe should be changed to For. 
Please consider revising the sentence on page 40, lines 20-25 of the Apphcant's 
disclosure. This sentence, as presently written, is grammatically awkward to the point 
that it is imclear. 

It is suggested that (Others) on page 41, line 23 of the Applicant's disclosure, be changed 
to (Miscellaneous), or something similar. 

A period (.) is missing on page 47, line 8 of the Apphcant's disclosure. 



Appropriate correction is required. 
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Claims 



Obiections 



Claims 42-44, 46, 48, 58-59, and 62-64 are objected to because of the following informalities. 



Claim 42-44, 46, 48, 58-59, and 62-64 recite the limitation "the images" or "said input images". There is 
insufficient antecedent basis for this limitation in the claim. For example, claim 44 recites, "said image 
input means performs image input where the images are captured at least two different points at different 
locations" and "color or density changes is performed between input images'". However, parent claim 3, 
recites, "image input means for inputting an image of said three dimensional object". That is, claim 3 
indicates input of a single image, whereas, claim 44 implies the input of at least one image. To rectify this, 
the AppHcant could change the language of the claims, in this case claim 3, to indicate that image input is 
of at least one image, as opposed to an image. Changes should be made to those claims of the set 42-44, 
46, 48, 58-59, and 62-64 that are independent. For the dependant claims (e.g. claim 44) of the set 42-44, 46, 
48, 58-59, and 62-64, changes should be made to the claims upon which these depend, (e.g. claim 3). 
Appropriate correction is required. 

10. Claims 5 and 17 are objected to because of the following informalities. The word exits should be 
replaced with the word exists. Appropriate action is required. 

Rejections Under 35 U.S.C. S 112(2) 

11. The following is a quotation of the second paragraph of 35 U.S.C. 112: 



12. Claims 49 and 64 rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for failing 
to particularly point out and distinctly claim the subject matter which appUcant regards as the invention. 



The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 
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13. These claims refer to divided regions of the input images. It is not clear to what divided regions of 
the input images the Applicant is referring. In accordance with their respective parent claims, divided 
regions of the input images, in claims 49 and 64 will be interpreted as portions where color density 
changes. This is consistent with the description found in the Applicant's specification of divided regions or 
strips. See the first paragraph on page 3 and the section, Detailed Shape Calculation Step Using Space 
Coding Method of the Applicant's specification. 

Rejections Under 35 U.S.C. S 112(1) 

14. The following is a quotation of the first paragraph of 35 U.S.C. 1 12: 



15. Claims 52 and 65 are rejected under 35 U.S.C. 1 12, first paragraph, as failing to comply with the 
enablement requirement. The claim(s) contains subject matter which was not described in the specification 
in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly 
connected, to make and/or use the invention. 

16. Claims 52 and 65 propose an apparatus and method, respectively, that include at least two three- 
dimensional shape inputting means or steps. The 3D modeling methods and apparatuses of the Applicant's 
specification, on the other hand, consist of at most two 3D shape inputting means or steps. As such, the 
Applicant's specification does not provide adequate support for the subject matter claimed in claims 52 and 



Rejections Under 35 U.S.C. S 102(b) 

17. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for 
the rejections under this section made in this Office action: 



(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in 
this country, more than one year prior to the date of application for patent in the United States. 



The specification shall contain a written description of the invention, and of the manner and process of making 
and using it, in such fiill, clear, concise, and exact terms as to enable any person skilled in the art to which it 
pertains, or with which it is most nearly connected, to make and use the same and shall set forth the best mode 
contemplated by the inventor of carrying out his invention. 
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18. Claims 52, 65-66 and 68-69 are rejected under 35 U.S.C. 102(b) as being anticipated by Niem 
("Error Analysis for Silhouette-Based 3D Shape Estimation from Multiple Views", WSNHC3DI 1997). 

19. The following is in regard to Claim 65. Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, from multiple camera views, using a shape-from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem further suggests extending this technique by 
incorporating a Structured Light (SL) methodology to accommodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 

(65.a.) Two three-dimensional shape inputting steps - e.g. a rough shape (SFS) input step and a 
detailed shape (SL) step - of obtaining the three dimensional shape of the three 
dimensional object. See, for example, the discussion below regarding claim 13. These 
steps (i.e. SFS and SL steps) have different accuracy in representing 3D objects. 
(65. b.) Capturing a 3D shape from multiple view points about the object, (e.g. by changing the 
position of moving the camera). See, for example, Niem Fig. 1(b). 
It has thus been shown that this method of Niem conforms to modeling method set forth in claim 65. 
Therefore, the method obtained by extending an SFS method to include an SL method, as per Niem's 
teachings, anticipates the modeling method set forth in claim 65. 

20. The following is in regard to Claim 52. Claims 52 recites substantially the same limitations as 
claim 65. (This claim proposes an apparatus implementing the method of claim 65). Therefore, with regard 
to claims 52, remarks analogous to those presented above with regard to claim 65 are apphcable. 

21. The following is in regard to Claim 68, Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, from multiple camera views, using sl shape from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem fiirther suggests extending this technique by 
incorporating a Structured Light (SL) methodology to accommodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 

(68.a.) Obtaining a rough shape information (i.e. an SFS representation) of the three dimensional 



object. 
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(68.b.) Obtaining a detailed shape information (i.e. an SL representation) of the three 
dimensional object. 

(68.C) Generating the three dimensional shape data of said three dimensional object based on 
the rough shape information and the detailed shape information. 
See Niem page 1, left colmnn, last paragraph to right colmnn, item (i). It has thus been shown that the 
method of Niem conforms to modeling method set forth in claim 68. Therefore, the method, obtained by 
extending an SFS method to include an SL method, as per Niem's teachings, anticipates the modeling 
method set forth in claim 68. 

22. The following is in regard to Claims 66 and 69. Claims 66 and 69 recite substantially the same 
limitations as claim 68. (These claims respectively propose an apparatus and stored computer program 
implementing the method of claim 68). Therefore, with regard to claims 66 and 69, remarks analogous to 
those presented above with regard to claim 68 are applicable. 



Rejections Under 35 U.S.C. § 103(a) 

23. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of 
this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject 
matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art 
to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made. 



24. Claims 1-4, 7, 10, 13-16, 19, 22, 25-28, 31, 34, 38-49, and 54-62 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Niem ("Error Analysis for Silhouette-Based 3D Shape Estimation from 
Multiple Views", WSNHC3DI 1997), in view of Battle, et al. ("Recent Progress in Coded Structure Light 
As a Technique to Solve the Correspondence Problem: A Survey", Pattern Recognition, Vol. 31, 1998). 



25. The following is in regard to Claim 13. Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, from multiple camera views, using a shape-from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem further suggests extending this technique by 



Application/Contn 
Art Unit: 2623 




imber: 09/847,011 




Page 10 



incorporating a Structured Light (SL) methodology to acconmiodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 

(13.b.) Inputting (i.e. capturing) an image(s) of the three dimensional object (e.g. input images - 

Niem page 1, left column, last paragraph). See, for example, Niem Fig. 1(b). 
(13.C.) Calculating a shape (e.g. a silhouette or the bounding volimie corresponding to obtained 
3D information) from the captured image(s). See Niem page 1, right column, last 
paragraph. 

(13.e.) Generating three dimensional shape data (e.g. the bounding volume) of the three 

dimensional object based on the shape. See Niem page 1, right column, last paragraph. 
While the step of (13.a) projecting pattern hght onto a three dimensional object is critical to SL methods, 
and, therefore, inherent to Niem's suggested extension of the aforementioned SFS method, Niem does not 
expressly demonstrate this step. SL methods also include the step of (13.d) calculating a shape from 
captured images, namely images of the projected pattern. Again, while this step would be inherent to the 
extended SFS of Niem, it is not explicitiy shown by Niem. 

26. Batile et al. disclose numerous Structured-Light methods to obtain 3D information to be used in 
the construction of 3D surfaces (presumably of objects - Batlle et al. Abstract). Generally speaking, 
Structiured-Light methods involve: 

(13.a.) Projecting pattern light onto a three dimensional object (i.e. "projecting a given pattern 

on the measuring surfaces" - Batlle et al. Abstract, lines 5-6). 
(13.b.) Inputting (i.e. capturing) an image(s) of the three dimensional object (Batile et al. page 

965, left colunm, lines 5-6). 
(13.d.) Calculating a shape (i.e. either the 2D deformed pattern projected on the Grabbed Image 
[see Batile et al. Fig. 2] plane, or the shape corresponding to obtained 3D information) 
from the captured image(s) (Batile et al. page 965, left column, lines 6-10). 
(13.e.) Generating three dimensional shape data of the three dimensional object based on the 

shape. See Batlle et al. page 965, left column, lines 6-10 and Batile et al. sections 4. 1 and 
4.3. 
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27. It is known that a silhouette-based representation of a 3D object does not accurately represent, or 
fails altogether, to represent concave surfaces of the object (Niem page 1, right column, item (-)), whereas, 
Structured-Light (SL) methods can accommodate such surfaces. In this manner, one can consider the 
silhouette-based representation (obtained by above step (13.c) of Niem) of an object a rough shape 
estimation of the object, relative to the representation obtained via the SL method (step (13.d) of Batlle et 
al.). Conversely, one can consider the representation obtained via the SL method a detailed shape 
estimation of the object, relative to a silhouette-based representation. Therefore, Niem suggests (Niem, 
page 1, right column, item (i)) calculating both a rough shape (i.e. silhouette-based model) and detailed 
shape (e.g. SL model such as those described by Batlle et al.), and using those shapes to generate three 
dimensional shape data of a three dimensional object. 

28. The teachings of Niem and Batlle et al. are combinable because they are analogous art. 
Specifically, the teachings of Niem and Batlle et al. are both directed toward the construction of a 3D 
model of an object from multiple views. Moreover, Niem provides an explicit suggestion of combinability 
(see above). Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
apphcant's claimed invention, to combine an SFS reconstruction method, in the manner suggested by Niem, 
with an SL method to obtain an accurate 3D representation of an object. The motivation to do so would 
have been to accurately represent surface concavities of the object that may otherwise have been missed by 
using an SFS technique alone. Extending an SFS method by combining it with a SL, in the maimer 
discussed above, yields a three dimensional modeling method that conforms to that which is set forth in 
claim 13. 

29. The following is in regard to Claims I and 25. Claims 1 and 25 recite substantially the same 
limitations as claim 13. (These claims respectively propose an apparatus and stored computer program 
implementing the method of claim 13). Therefore, with regard to claims 1 and 25, remarks analogous to 
those presented above with regard to claim 13 are applicable. 

30. The following is in regard to Claims 2, 14, and 26. Note that these claims deviate from claims 1, 
13, and 25, respectively, only in that the aforementioned rough shape is input or captured, as opposed to 
being calculated. This does not introduce anything significant over claims 1,13 and 25, respectively, since 
the calculated rough shape data may just as well be regarded as input data. In this way, claims 2 , 14 and 26 
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can be regarded as reciting substantially the same limitations as claim 1,13, and 25, respectively. 
Therefore, with regard to claims 2, 14 and 26, remarks analogous to those presented above with regard to 
claim 1, 13, and 25 are respectively applicable. 

3 1 . The following is in regard to Claim 1 P. As shown above, the teachings of Niem and Batlle et al. , 
when combined in maimer discussed above, adequately satisfy the limitations of claim 13. Inherent to the 
step of calculating the detailed shape (i.e. the SL reconstruction), is acquiring a plurality of object surface 
location candidates from the input image. Since the SL method (step (13.d) above) is used to refine the SFS 
model (step (13.c) above), one can reasonably consider all points on an within the SFS model (i.e. the 
rough 3D shape) as location candidates. This, in turn, implies that points in the input image(s) that he 
within silhouettes of the object can represent location candidates, from which the detailed shape can be 
derived. Moreover, since the depth of all points in the captured image(s) is calculated, every point within 
the captured image could also be regarded as location candidates. In this way, extending an SFS method by 
combining it with a SL, in the manner discussed above, yields a three dimensional modeling method that 
conforms to that which is set forth in claim 19. 

32. The following is in regard to Claim 7 and 31. Claims 7 and 31 recite substantially the same 
limitations as claim 19. (These claims respectively propose an apparatus and stored computer program 
implementing the method of claim 19). Therefore, with regard to claims 7 and 3 1, remarks analogous to 
those presented above with regard to claim 19 are applicable. 

33. The following is in regard to Claim 22. As shown above, the teachings of Niem and Batlle et al., 
when combined in maimer discussed above, adequately satisfy the limitations of claim 13. It was also 
discussed above that the SFS (shape-from-silhouette) method is used to derive the rough shape of the 
reconstruction proposed by Niem. The SFS method uses silhouettes of the object to obtain a 3D model of 
that object. See Fig. 1(b) of Niem. Therefore, extending an SFS method by combining it with a SL, in the 
maimer discussed above, yields a three dimensional modeling method that conforms to that which is set 
forth in claim 22. 

34. The following is in regard to 10 and 34. Claims 10 and 34 recite substantially the same limitations 
as claim 22. (These claims respectively propose an apparatus and stored computer program implementing 
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the method of claim 22). Therefore, with regard to claims 10 and 34, remarks analogous to those presented 
above with regard to claim 22 are applicable. 

35, The following is in regard to Claim 15. Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, from multiple camera views, using a shape-from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem fiirther suggests extending this technique by 
incorporating a Structured Light (SL) methodology to accommodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 

(IS.aO Inputting (i.e. capturing) an image(s) of the three dimensional object (e.g. input images - 
Niem page 1, left column, last paragraph), said image(s) being obtained by projecting 
light (i.e. illmninating) the three dimensional object. See, for example, Niem Fig. 1(b). 

( 1 5 .b. ) C^culating a shape (e. g. a silhouette or the or the bounding volume corresponding to 
obtained 3D information) from the captured image(s). See Niem page 1, right column, 
last paragraph. 

(15.d,) Generating three dimensional shape data (e.g. the bounding volume) of the three 

dimensional object based on the shape. See Niem page 1, right colixmn, last paragraph. 
SL methods also include the step of (15.c) calculating a shape from captured images, namely images of the 
projected pattern. Again, while this step would be inherent to the extended SFS of Niem, it is not explicitly 
shown by Niem. Lastly, Niem does not expUcitly show that the input image(s) be obtained by projecting 
pattern light onto said three dimensional image(s) (though, as will be discussed, below this aspect would be 
implicit to a combination of the SFS and SL methods). 

36. Batlle et al. disclose numerous Structured-Light methods to obtain 3D information to be used in 
the construction of 3D surfaces (presmnably of objects - Batlle et al. Abstract). Generally speaking, 
Structured-Light methods involve: 

(15.a2) Inputting (i.e. capturing) an image(s) of the three dimensional object, said image(s) 

obtained by projecting pattern light onto the three dimensional object (i.e. "projecting a 
given pattern on the measuring surfaces" and "imag[ing] the illuminated scene" - Batlle 
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et al. Abstract, lines 4-6). 
(15.C.) Calculating a shape (i.e. either the 2D deformed pattern projected on the Grabbed Image 

[see Batlle et al. Fig. 2] plane, or the shape corresponding to obtained 3D information) 

from the captured iniage(s) (Batlle et al. page 965, left column, lines 6-10). 
(15.d.) Generating three dimensional shape data of the three dimensional object based on the 

shape. See Batlle et al. page 965, left column, lines 6-10 and Batlle et al. sections 4. 1 and 

4.3. 

37. For the reasons given above, one can consider the silhouette-based representation (obtained by 
above step (15.c) of Niem) of an object a rough shape estimation of the object, relative to the 
representation obtained via the SL method (step (15.d of Batlle et al.)). Conversely, one can consider the 
representation obtained via the SL method a detailed shape estimation of the object, relative to a silhouette- 
based representation. Therefore, Niem suggests (Niem, page 1, right column, item (i)) calculating both a 
rough shape (i.e. silhouette-based model) and detailed shape (e.g. SL model such as those described by 
Batile et al.), and using those shapes to generate three dimensional shape data of a three dimensional object. 

38. Though neither Niem nor Batile et al. explicitiy suggest that the rough shape and detailed shape be 
obtained by the same image(s) (i.e. the image(s) of the object illuminated by the projected pattern), this 
aspect would be inherent to Niem's suggested combination of the SFS and SL methods. Since the SL 
method consists of projecting a pattern onto a three dimensional object, Niem's suggested combination of 
the SFS and SL methods entails the capture of an image(s) of the object illuminated by the projected 
pattern. As further suggested by Niem, the silhouette can be obtained by chroma-keying or color-keying 
(e.g. blue-screening - Niem page 1, left column, last paragraph), which can acquire the silhouette of the 
object even witii a light pattern projected thereon^. Therefore, it is reasonable to assume that, in a method 
that combines the SFS and SL methodologies, the image(s), from which the rough shape (i.e. the silhouette- 
based representation) and the detailed shape (e.g. an SL model) are derived, is the same (i.e. the image(s) in 
steps (15.ai) and (15.a2) can be considered the same). 



2 Capturing separate images under different lighting conditions is impractical and even undesirable since it introduces unnecessary 
processing and complexity to the method and, moreover, may introduce inconsistency between the captured images. 
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39. The teachings of Niem and Batlle et al. are combinable because they are analogous art. 
Specifically, the teachings of Niem and Batlle et al. are both directed toward the construction of a 3D 
model of an object from multiple views. Moreover, Niem provides an explicit suggestion of combinability 
(see above). Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
appUcant's claimed invention, to combine an SFS reconstruction method, in the manner suggested by Niem, 
with an SL method to obtain an acciuate 3D representation of an object. The motivation to do so would 
have been to accurately represent surface concavities of the object that may otherwise have been missed by 
using an SFS technique alone. Extending an SFS method by combining it with a SL, in the manner 
discussed above, yields a three dimensional modeling method that conforms to that which is set forth in 
claim 15. 

40. The following is in regard to Claim 3 and 27. Claims 3 and 27 recite substantially the same 
limitations as claim 15. (These claims respectively propose an apparatus and stored computer program 
implementing the method of claim 15). Therefore, with regard to claims 3 and 27, remarks analogous to 
those presented above with regard to claim 15 are applicable. 

41. The following is in regard to Claims 4, 16, and 28. Note that these claims deviate from claims 3, 
15, and 27, respectively, only in that the aforementioned rough shape is input or captured, as opposed to 
being calculated. This does not introduce anything significant over claims 3,15 and 27, respectively, since - 
the calculated rough shape data may just as well be regarded as input data. In this way, claims 4 , 16 and 28 
can be regarded as reciting substantially the same limitations as claim 3,15, and 27, respectively. 
Therefore, with regard to claims 4, 16 and 28, remarks analogous to those presented above with regard to 
claim 3, 15, and 27 are respectively applicable, 

42. The following is in regard to Claim 44. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 3. Generally, 3D 
reconstruction flrom captured images involves: 

(44.a.) Image input is performed such that the images are captured at least two different points at 
different locations. For example, the SFS method necessitates this. See Niem page 1, Fig. 
1(b) and left colunm, last paragraph. 
Furthermore, coded structured Ught (CSL) reconstruction involves: 
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(44.b.) Detailed shape (see above) calculation entails extracting a portion in said input image 

where color or density changes. Refer to, for instance, Batlle et al. Fig. 3 (page 969) and 
the corresponding description in Batlle et al. Section 6. 1. The various patterns are 
projected onto the scene. Notice these patterns consist of color or density changes and are 
detected by the camera, during the 3D scene reconstruction. Also, notice that temporal 
changes in the color or density of the projected patterns are also observed during 
reconstruction (paragraph 2 of Batile et al. Section 6. 1). 

(44.C.) The detailed shape is derived by matching of potions where color or density changes is 
performed between input images, to calculate the three dimensional shape of said three 
dimensional object. This can be imderstood from Batile et al. page 963, lines 4-9 and 
lines 6-8 of the Abstract. Color or density changes are encoded for each imaged region 
and, from the obtained codes, a correspondence (matching) is derived between 
corresponding regions of the captured image and the projected image (Batlle et al. page 
965, Fig. 2). 

A CSL methodology has the advantage that it greatly simplifies the correspondence problem (Batile et al. 
page 963, right column, lines 4-5). It would, therefore, have been obvious to one of ordinary skill in the art, 
at the time of the applicant's claimed invention, to use a CSL methodology (e.g. Posdamer et al.'s CSL 
method - Batile et al. Section 6.1), as an SL method in Niem's suggested combination of the SFS and SL 
methodologies. In this way, an apparatus implementing the method obtained by combining the SFS and 
CSL methods, as proposed by Niem and discussed above, would conform to the three dimensional 
modeling apparatus set forth in claim 15. 

43. The following is in regard to Claim 54. Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, fi*om multiple camera views, using a shape-from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem further suggests extending this technique by 
incorporating a Structured Light (SL) methodology to accommodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 
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(54.b.) Inputting (i.e. capturing) an image(s) of the three dimensional object (e.g. input images - 
Niem page 1, left column, last paragraph). See, for example, Niem Fig. 1(b). 

(54.ci ) Calculating a shape (e.g. a silhouette or the or the bounding volume corresponding to 
obtained 3D information) from the captured image(s). See Niem page 1, right colimm, 
last paragraph. 

44. While the step of (54.a) projecting pattern Ught onto a three dimensional object is critical to SL 
methods, and, therefore, inherent to Niem's suggested extension of the aforementioned SFS method, Niem 
does not expressly demonstrate this step. SL methods also include the step of (54.C2) calculating a shape 
from captured images, by at least extracting a pattern projected region and pattern border region the said 
input image(s) to calculate the three dimensional shape of said three dimensional object based on these 
regions. Again, while this step would be inherent to the extended SFS of Niem, it is not explicitly shown by 
Niem. 

45. Batlle et al. disclose numerous Structured-Light methods to obtain 3D information to be used in 
the construction of 3D surfaces (presumably of objects - Batlle et al. Abstract). Generally speaking, 
Structured-Light methods involve: 

(54.a.) Projecting pattern light onto a three dimensional object (i.e. "projecting a given pattern 
on the measuring surfaces" - Batlle et al. Abstract, lines 5-6). 

(54.b.) Inputting (i.e. capturing) an image(s) of the three dimensional object (Batlle et al. page 
965, left colunrn, lines 5-6). 

(54.C2) Calculating a shape (i.e. either the 2D deformed pattern projected on tiie Grabbed Image 
[see Batlle et al. Fig. 2] plane, or the shape corresponding to obtained 3D information) 
from the captured image(s) (Batlle et al. page 965, left column, lines 6-10), wherein 
calculating a shape from captured images involves at least extracting a pattern projected 
region (e.g. the patterns depicted in Batlle et al. Fig. 3) and pattern border region (e.g. the 
borders between the light and dark regions of the patterns depicted in Batlle et al. Fig. 3) 
the said input image(s) to calculate the three dimensional shape of said three dimensional 
object based on these regions. See Batlle et al. page 965, left column, lines 6-10 and 
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Batlle et al. sections 4.1 and 4.3. 



46. The teachings of Niem and Batlle et al. are combinable because they are analogous art. 
Specifically, the teachings of Niem and BatUe et al. are both directed toward the construction of a 3D 
model of an object from multiple views. Moreover, Niem provides an expUcit suggestion of combinability 
(see above). Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
apphcant*s claimed invention, to combine an SFS reconstruction method, in the manner suggested by Niem, 
with an SL method to obtain an accurate 3D representation of an object. The motivation to do so would 
have been to accurately represent surface concavities of the object that may otherwise have been missed by 
using an SFS technique alone. Extending an SFS method by combining it with a SL, in the manner 
discussed above, yields a three dimensional modeling method that conforms to that which is set forth in 
claim 54. 

47. The following is in regard to Claim 56 . As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 54. Fiuthermore, 
Batlle et al. suggest the usage of a plurality of binary patterns as the projected pattern hght See Batlle et al. 
Fig. 3 and the second paragraph (first sentence) of Batlle et al. Section 6. 1. In this way, the method 
obtained by combining the SFS and CSL methods, as proposed by Niem and discussed above, would 
conform to the three dimensional modeling method set forth in claim 56. 

48. The following is in regard to Claim 57. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 54. Furthermore, 
Batlle et al. suggest the capture of image from at least two different locations. See, for example, the 
Introduction of Batlle et al., left column, lines 5-8. Moreover, it should be clear that repeating the SL 
reconstruction for multiple viewpoints about the object is necessary to obtain a full 3D representation of the 
object. In this way, the method obtained by combining the SFS and CSL methods, as proposed by Niem 
and discussed above, would conform to the three dimensional modeling method set forth in claim 57. 

49. The following is in regard to Claims 38, 40 and 41. Claims 38 and 40-41 recite substantially the 
same hmitations as claims 54 and 56-57, respectively. (These claims propose an apparatus implementing 
the method of claim 54 and 56-57, respectively). Therefore, with regard to claims 38 and 40-41 remarks 
analogous to those presented above with regard to claims 54 and 56-57 are respectively applicable. 
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50. The following is in regard to Claims 39. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 1. Note that, when 
taking into account the subject matter inherited from parent claim 1, claim 39 puts forth substantially the 
same hmitations as claim 38. Therefore, with regard to claim 39 remarks analogous to those presented 
above with regard to claim 38 are applicable. 

5 1 . The following is in regard to Claims 55. As shown above, the teachings of Niem and Batlle et al. , 
when combined in manner discussed above, adequately satisfy the limitations of claim 1. Note that, when 
taking into account the subject matter inherited from parent claim 13, claim 55 puts forth substantially the 
same Hmitations as claim 54. Therefore, with regard to claim 55 remarks analogous to those presented 
above with regard to claim 54 are applicable. 



52. The following is in regard to Claim 58. Niem discloses a robust approach for the automatic 3D 
reconstruction of an object, from multiple camera views, using a shape-from-silhouette (SFS) technique 
(Niem page 1, left column, last paragraph). Niem fiirther suggests extending this technique by 
incorporating a Structured Light (SL) methodology to acconunodate the concave surfaces of the observed 
object (Niem, page 1, right column, item (i)). This extended method involves: 

(58.b.) Inputting (i.e. capturing) an image(s) of the three dimensional object (e.g. input images - 
Niem page 1, left column, last paragraph). See, for example, Niem Fig. 1(b). 

(58.Ci) Calculating a shape (e.g. a silhouette or the or the bounding volume corresponding to 
obtained 3D information) from the captured image(s). See Niem page 1, right column, 
last paragraph. 

53. While the step of (58.a) projecting pattern hght onto a three dimensional object is critical to SL 
methods, and, therefore, inherent to Niem's suggested extension of the aforementioned SFS method, Niem 
does not expressly demonstrate this step. SL methods also include the step of (58.C2) calculating a shape 
from captured images, by: 

(58.C2. 1) Extracting a portion in said input image where color or density changes. 
(58.C2.2) Matching portions where color or density changes between input images. 
Again, while this step would be inherent to the extended SFS of Niem, it is not explicitly shown by Niem. 
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54. Batlle et al. disclose numerous Structured-Light methods to obtain 3D information to be used in 
the construction of 3D surfaces (presumably of objects - Batile et al. Abstract). Generally speaking, 
Structured-Light methods involve: 

(54.a.) Projecting pattern light onto a three dimensional object (i.e. "projecting a given pattern 

on the measuring surfaces" - Batlle et al. Abstract, lines 5-6). 
(54.b.) Inputting (i.e. capturing) an iniage(s) of the three dimensional object (Batlle et al, page 

965, left column, lines 5-6). 
(54.C2) Calculating a shape (i.e. either the 2D deformed pattern projected on the Grabbed Image 
[see Batlle et al. Fig. 2] plane, or the shape corresponding to obtained 3D information) 
from the captured image(s) (Batlle et al. page 965, left column, lines 6-10). 
Step (54.C2) includes: 

(58.C2. 1) Extracting a portion in said input image where color or density changes. See, for 



(58.C2.2) Matching portions where color or density changes (e.g. spatio-temporal color/density 



Furthermore, Batlle et al. suggest the capture of image (step (58.b) above) from at least two different 
locations. See, for example, the Introduction of Batlle et al., left column, hues 5-8. Moreover, it should be 
clear that repeating the SL reconstruction for multiple viewpoints about the object is necessary to obtain a 
fixll 3D representation of the object. 

55. The teachings of Niem and Batlle et al. are combinable because they are analogous art. 
Specifically, the teachings of Niem and Batlle et al. are both directed toward the construction of a 3D 
model of an object from multiple views. Moreover, Niem provides an expUcit suggestion of combinability 
(see above). Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's claimed invention, to combine an SFS reconstruction method, in the manner suggested by Niem, 
with an SL method to obtain an accurate 3D representation of an object. The motivation to do so would 
have been to accurately represent surface concavities of the object that may otherwise have been missed by 



example, Batlle et al. Section 3, paragraph 1 and Figs. 3-4. 



changes encapsulated by the codification of the projected patterns) between input 



images. See, for example, Batlle et al. Section 3, Fig. 2, paragraphs 2-5 of Section 5, 



and page 972, right column, paragraph 1. 
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using an SFS technique alone. Extending an SFS method by combining it with a SL, in the manner 
discussed above, yields a three dimensional modeling method that conforms to that which is set forth in 
claim 58. 

56. The following is in regard to Claim 60. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 58. Batlle et al. 
suggest that color structured patterns can be projected onto a object or scene. Batlle et al. fiirther suggest 
that hue be used for the color pattern generation, thus implying the usage of a color model using hue (e.g. 
HSI, HSV, etc). See Batlle et al. Section 5, Colour. Furthermore, Batlle et al. suggest that the correlation 
between adjacent pattern elements be small so that strong contrast exists between at element borders (i.e. 
"the hue values have to be quite different from each other"). Adjacent stripes having a hue-shift of 90° 
would easily satisfy this hmitation. See Batlle et al. Section 5, Colour and paragraph 1 of section 6.8 (page 
976). 

57. Given the teachings of Batlle et al. It would have been obvious to one of ordinary skill in the art, 
at the time of the applicant's claimed invention, to utilize a color SL pattern, consisting of relatively 
uncorrelated hues, as the pattern projected onto the object undergoing 3D reconstruction. One clear 
advantage of using a color patterns, in CSL schemes such as those presented by Batlle et al., is that the code 
basis is extended beyond two symbols (e.g. colors), as in binary. This allows more information (more 
codes) to be presented per captured image. Using a color SL pattern in the method obtained by combining 
the SFS and CSL methods, as proposed by Niem and discussed above, yields a modeling method that 
conform to the three dimensional modeling method set forth in claim 60. 

58. The following is in regard to Claim 61. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 58. Generally, in 
coded SL methods, matching of the portions where color or density changes (e.g. spatio-temporal 
color/density changes encapsulated by the codification of the projected pattems ) is obtained based on 
properties of the portions where color or density changes, which are obtained from the input image(s). See 
Batlle et al. Fig. 2. In this way, the method obtained by combining the SFS and CSL methods, as proposed 
by Niem and discussed above, would conform to the three dimensional modeling method set forth in claim 
61. 
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59. The following is in regard to Claim 62. As shovm above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 61. Generally, in 
coded SL methods, the property of the portion where color or density changes is color information 
regarding portions located to the left and right of, or above and below, said portion where color or density 
changes in said input images. See Batlle et al. Fig. 2. In this way, the method obtained by combining the 
SFS and CSL methods, as proposed by Niem and discussed above, would conform to the three 
dimensional modeling method set forth in claim 62. 

60. The following is in regard to 42 and 45-47, Claims 42 and 45-47 recite substantially the same 
limitations as claims 58 and 60-62, respectively. (These claims propose an apparatus implementing the 
method of claims 58 and 60-62, respectively). Therefore, with regard to claims 42 and 45-47 remaiks 
analogous to those presented above with regard to claims 58 and 60-62 are respectively applicable. 

61. The following is in regard to Claim 43. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 1. Note that when 
taking into account the subject matter inherited from parent claim 1, claim 43 puts forth substantially the 
same limitations as claim 42. Therefore, with regard to claim 43 remarks analogous to those presented 
above with regard to claim 42 are applicable. 

62. The following is in regard to Claim 59. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 13. Note that when 
taking into account the subject matter inherited from parent claim 13, claim 59 puts forth substantially the 
same hmitations as claim 58. Therefore, with regard to claim 59 remarks analogous to those presented 
above with regard to claim 58 are applicable. 



63. Claims 5-6, 8-9, 17-18, 20-21, 29-30, and 32-33 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Niem, in view of Batlle, et al., in further view of Wojciech et al. ("Image-Based Visual 
Hulls", SIGGRAPH 2000). 



64. The following is in regard to Claim 1 7. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 13. Though it can be 
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argued that, in the extended SFS/SL method of Niem and Batlle et al, the step of generating three 
dimensional shape data inherently includes the following aspects: 

(17.a.) Determining the final three dimensional shape data based on the following rule: 
(17.b.) If the detailed shape exists inside the rough shape, then the detailed shape is the 

final three dimensional shape data of said object 
(17.C.) Else the rough shape is taken as the final three dimensional shape data of said 

object 

these aspects are not expressly disclosed by either Niem or Batlle et al. 

65. Wojciech et al. disclose an SFS method employing the concept of visual hulls. A visual hull is 
volumetric representation formed essentially of the intersections of silhouette cones. It will always contain 
the object. Refer to Visual Hull in Section 2 and Fig. 1 of Wojciech et al. Again concave surfaces cannot be 
accounted for (last sentence of paragraph 2 of Visual Hull in Section 2 of Wojciech et al.). In this maimer, 
the visual hull represents a rough shape, as discussed before. Wojciech et al. refine this model by "carving" 
away (calculatus eliminatus - Wojciech et al. page 1, right column paragraph 1) regions inside the visual 
hull where the object is not. In other words, Wojciech et al. obtains a model of an object such that: 

(17.b.) If the detailed shape (i.e. the carved-away model) exists inside the rough shape, 

then the detailed shape is the final three dimensional shape data of said object. 

(17.C.) Else the rough shape (i.e. the visual hull) is taken as the final three dimensional 

shape data of said object. 
This observation follows from the carving away process discussed above. 

66. Wojciech et al. is combinable with the Niem and Batlle et al. because they are analogous art. 
Specifically, Wojciech et al. teaches an SFS method and Niem teaches using an SFS method in conjunction 
with an SL method (e.g. those taught by Batlle et al.). Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to use the visual hull method of 
Wojciech et al. for SFS reconstruction in the method obtained by combining the SFS and SL methods in 
accordance with the teachings of Niem and Batlle et al. Furthermore, having the SL framework at his/her 
disposal, it would have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to use an SL reconstructed model as the detailed shape in step (17.b) above. The motivation to 
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use the visual hull method of Wojciech et al. as the SFS method would have been that it overcomes 
computational expense of CSG/volumetric shape-from-silhouette methods discussed by Niem (Niem page 
1, left column, paragraph 1, sentence 1). See Wojciech et al. page 1, right coliunn, paragraph 2. The 
motivation to use the SL derived model as the detailed shape would have been that it would replace the 
expensive carving operation suggested by Wojciech et al. with a simple decision as to whether a voxel lies 
within the visual hull (e.g. steps (17.b)-(17.c) above). Combining the teachings of Wojciech et al., Batlle et 
al. and Niem, in the manner just described, would yield a modeling method that conforms to that which is 
set forth in claim 17. 

67. The following is in regard to Claim 18. As shown above, the teachings of Niem and Batlle et al., 
when combined in maimer discussed above, adequately satisfy the limitations of claim 13. Though it can be 
argued that, in the extended SFS/SL method of Niem and Batlle et al, the step of generating three 
dimensional shape data inherently includes obtaining a region where the object surface can exist based on 
the rough shape, to generate three dimensional shape data of said three dimensional object based on the 
rough shape, the detailed shape, and the region where the object surface can exist, this is not expUcitly 
shown by either Niem or Batlle et al. 

68. Wojciech et al. disclose an SFS method employing the concept of visual hulls, A visual hull is 
volumetric representation formed essentially of the intersections of silhouette cones. It will always contain 
the object. Refer to Visual Hull in Section 2 and Fig. 1 of Wojciech et al. Again concave surfaces cannot be 
accounted for (last sentence of paragraph 2 of Visual Hull in Section 2 of Wojciech et al.). In this manner, 
the visual hull represents a rough shape, as discussed before. Wojciech et al. refine this model by "carving" 
away (calculatus eliminatus - Wojciech et al. page 1, right column paragraph 1) regions inside the visual 
hull where the object is not. In other words, the surface and inside of the visual hull represents a "region 
where the object surface can exist based on the rough shape". Therefore, 3D reconstruction, according to 
Wojciech et al.'s visual hull method, is based on a rough shape (i.e. the visual hull) and a region where the 
object surface can exist (all points within and on the visual hull). 

69. Wojciech et al. is combinable with the Niem and Batlle et al. because they are analogous art. 
Specifically, Wojciech et al. teaches an SFS method and Niem teaches using an SFS method in conjunction 
with an SL method (e.g. those taught by Batlle et al.). Therefore, it would have been obvious to one of 
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ordinary skill in the art, at the time of the applicant's claimed invention, to use the visual hull method of 
Wojciech et al. for SFS reconstruction in the method obtained by combining the SFS and SL methods in 
accordance with the teachings of Niem and Batlle et al. The motivation to use the visual hull method of 
Wojciech et al. as the SFS method would have been that it overcomes computational expense of 
CSG/volimietric shape-from-silhouette methods discussed by Niem (Niem page 1, left column, paragraph 
1, sentence 1). See Wojciech et al. page 1, right column, paragraph 2. Recall that, in the method obtained 
by extending an SFS method by combining it with a SL, in the manner discussed above and by Niem, three 
dimensional shape data of the object is generated based on a rough shape (SFS reconstruction) and detailed 
shape (SL reconstruction. Therefore, by combining the method of Niem and Batlle et al. with that of 
Wojciech et al., in the maimer just described, one obtains a method, in accordance with claim 13, wherein 
generating three dimensional shape data includes obtaining a region where the object surface can exist 
based on the rough shape, to generate three dimensional shape data of said three dimensional object based 
on the rough shape, the detailed shape, and the region where the object surface can exist. 

70. The following is in regard to Claim 20. As shown above, the teachings of Niem and Batlle et al., 
when combined in maimer discussed above, adequately satisfy the limitations of claim 13. Though it can be 
argued that, in the extended SFS/SL method of Niem and Batlle et al, the step of calculating a detailed 
shape includes setting a region where the object surface cannot exist in a portion inside the rough shape and 
determining a region inside the rough shape other than said region where the object surface cannot exist as 
a region where the object surface can exist, to generate three dimensional shape data of said three 
dimensional object based on the rough shape, the detailed shape, and the region where the object surface 
can exist, neither Niem nor Batlle et al. explicitly show such a step. 

71. Wojciech et al. disclose an SFS method employing the concept of visual hulls. A visual hull is 
volumetric representation formed essentially of the intersections of silhouette cones. It will always contain 
the object. Refer to Visual Hull in Section 2 and Fig. 1 of Wojciech et al. Again concave surfaces cannot be 
accounted for (last sentence of paragraph 2 of Visual Hull in Section 2 of Wojciech et al.). In this manner, 
the visual hull represents a rough shape, as discussed before. 

72. Wojciech et al. refine this model by "carving" away {calculatus eliminatus - Wojciech et al. page 
1, right column paragraph 1) regions inside the visual hull where the object is not. Observing a captured 
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iinage(s), Wojciech et al. determines whether pixels in the image(s) correspond to visible regions of the 
object or regions that are occluded by the visual hull. See Wojciech et al. Fig. 4 and page 4, left column, 
last paragraph and right column, first paragraph. Also refer to Wojciech et al. page 5, paragraphs 1-3. Self- 
occluding regions (e.g. regions occluded by the actual (unknown) geometry of the object or the visual hull) 
are marked as not visible. Points that are not occluded are marked as visible. See, for example, Wojciech et 
al. page 4, Fig. 4, right column paragraph 1, and the pseudocode in the right column. Wojciech et al.'s 
visual hull method, therefore, includes setting a region where the object surface cannot exist (i.e. "not 
visible" marked regions) and a portion inside the rough shape (i.e. the visual hull) and determining a region 
inside the rough shape other than said region where the object surface cannot exist as a region where the 
object surface can exist (i.e. "visible" marked regions). 

73. Wojciech et al. is combinable with the Niem and Badle et al. because they are analogous art. 
Specifically, Wojciech et al. teaches an SFS method and Niem teaches using an SFS method in conjimction 
with an SL method (e.g. those taught by Batlle et al.). Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to use the visual hull method of 
Wojciech et al. for SFS reconstruction in the method obtained by combining the SFS and SL methods in 
accordance with the teachings of Niem and Batlle et al. The motivation to use the visual hull method of 
Wojciech et al. as the SFS method would have been that it overcomes computational expense of 
CSGA^olumetric shape-from-silhouette methods discussed by Niem (Niem page 1, left column, paragraph 
1, sentence 1). See Wojciech et al. page 1, right column, paragraph 2. It would have also been obvious to 
one of ordinary skill in the art, at the time of the applicant's claimed invention, to use points that are marked 
visible (i.e. existent regions) to generate three dimensional shape data of the three dimensional object. 
Clearly the motivation for this would have been to constrain the model only those points that are visible in 
some view (as opposed to including points that are not visible in all views), thereby, eliminating extraneous 
points from the model and/or model computation. Combining the teachings of Wojciech et al., Niem and 
Batlle et al., in the manner just described, would yield a modeling method that conforms to the 3D 
modeling method set forth in claim 20. 

74. The following is in regard to Claim 21. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 13. Though it can be 
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argued that, in the extended SFS/SL method of Niem and Batlle et al., the step of generating three 
dimensional shape data includes determining a final object surface location candidate firom a plurality of 
object surface location candidates based on a region where the object surface can exist and generating three 
dimensional shape data of said three dimensional object based on said final object surface location 
candidate, this step is not explicitly shown by either of Niem or Batlle et al. 

75. Wojciech et al. disclose an SFS method employing the concept of visual hulls. A visual hull is 
volumetric representation formed essentially of the intersections of silhouette cones. It will always contain 
the object. Refer to Visual Hull in Section 2 and Fig. 1 of Wojciech et al. Again concave surfaces cannot be 
accounted for (last sentence of paragraph 2 of Visual Hull in Section 2 of Wojciech et al.). In this manner, 
the visual hull represents a rough shape, as discussed before. Wojciech et al. refine this model by "carving" 
away (calculatus eliminatus - Wojciech et al. page 1, right column paragraph 1) regions inside the visual 
hull where the object is not. In other words, the surface and inside of the visual hull represents a "region 
where the object surface can exist". Points inside the visual hull would therefore constitute surface location 
candidates. (Points that are marked visible [Wojciech et al. page 4, right column paragraph 1] could also be 
considered location candidates). 

76. Wojciech et al. is combinable with the Niem and Batlle et al. because they are analogous art. 
Specifically, Wojciech et al. teaches an SFS method and Niem teaches using an SFS method in conjimction 
with an SL method (e.g. those taught by Batlle et al.). Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to use the visual hull method of 
Wojciech et al. for SFS reconstruction in the method obtained by combining the SFS and SL methods in 
accordance with the teachings of Niem and Batlle et al. The motivation to use the visual hull method of 
Wojciech et al. as the SFS method would have been that it overcomes computational expense of 
CSG/volumetric shape-fi-om-silhouette methods discussed by Niem (Niem page 1, left column, paragraph 
1, sentence 1). See Wojciech et al. page 1, right column, paragraph 2. Given the discussion above, it should 
be clear that combining the teachings of Wojciech et al. with those of Niem and Batlle et al., yields a 
method, in accordance with claim 13, wherein the step of generating three dimensional shape data includes 
determining a final object surface location candidates (i.e. points composing the SL reconstruction of the 
object) from a plurahty of object surface location candidates (i.e. points on and within the visual hull) based 
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on a region (i.e. the visual hull) where the object surface can exist and generating three dimensional shape 
data of said three dimensional object based on the final object surface location candidates. Such a method is 
in accordance with claim 21. 



limitations as claims 17-18 and 20-21, respectively. (These claims respectively propose an apparatus 
implementing, respectively, the methods of claims 17-18 and 20-21). Therefore, with regard to claims 5-6 
and 8-9, remarks analogous to those presented above with regard to claim 17-18 and 20-21 are respectively 
apphcable. 

78. The following is in regard to Claims 29-30 and 32-33. Claims 29-30 and 32-33 recite substantially 
the same Umitations as claims 17-18 and 20-21, respectively. (These claims respectively propose a stored 
computer program implementing, respectively, the methods of claims 17-18 and 20-21). Therefore, with 
regard to claims 29-30 and 32-33, remarks analogous to those presented above with regard to claim 17-18 
and 20-21 are respectively applicable. 

79. Claims 63-64 are rejected under 35 U.S.C. 103(a) as being unpatentable over Niem, in view of 
Batlle, et al., in further view of Gonzalez et al. ("Digital Image Processing", Irst Edition). 

80. The following is in regard to Claim 63. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 61. Neither Niem 
nor Batlle et al. show averaging being performed on the input images when said property is obtained. 

81. Image averaging is known to reduce noise in images. See Section 4.2.4 of Gonzalez et al. 

82. The teachings of Gonzalez et al. are combinable with those of Niem and Batlle et al. because they 
are analogous art. Specifically, each concern processing of digital images captured of an object. Therefore, 
it would have been obvious to one of ordinary skill in the art, at the time, of the appUcant's claimed 
invention, to average the input images when said property is obtained. The motivation to do so would have 
been to reduce noise, thereby improving the accuracy with which the correspondence between images can 
be derived. Incorporating image averaging into the method obtained by combining the SFS and CSL 



77. 



The following is in regard to Claims 5-6 and 8-9. Claims 5-6 and 8-9 recite substantially the same 
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methods, as proposed by Niem and discussed above, would yield a modeling method that conforms to the 
three dimensional modeling method set forth in claim 63. 

83 . The following is in regard to Claim 64. As shown above, the teachings of Niem and Batlle et al. , 
when combined in manner discussed above, adequately satisfy the limitations of claim 63. Clearly, 
averaging an input image is equivalent to averaging each of the divided regions of the image, and vice 
versa. It should also be clear from Section 4.2.4 of Gonzalez et al. that arbitrary sections of an image may 
be averaged. Taking this into account, the discussion above with regard to claim 63 sufficiently addresses 
the subject matter of claim 63. Therefore, incorporating image averaging into the method obtained by 
combining the SFS and CSL methods, as proposed by Niem and discussed above, would yield a modeling 
method that conforms to the three dimensional modeling method set forth in claim 64. 

84. The following is in regard to 48-49. Claims 48-49 recite substantially the same limitations as 
claims 63-64, respectively. (These claims propose an apparatus implementing the method of claims 58 and 
60-62, respectively). Therefore, with regard to claims 48-49 remarks analogous to those presented above 
with regard to claims 63-64 are respectively applicable. 



85. Claims 53 and 67 are rejected under 35 U.S.C. 103(a) as being unpatentable over Niem, in view of 
Crampton (U.S. Patent 6,611,617). 



86. The following is in regard to Claim 67, As shown above, Niem discloses an apparatus that 
conforms to that which is set forth in claim 66. Niem, however, does not expressly show or suggest 
controlling the position of the second three dimensional shape measuring means, said control means being 
controlled based on said rough shape information obtained by said first three dimensional measuring 
means. 

87. Crampton discloses an SL scanning apparatus for obtaining 3D models of scanned objects. The 
scanning apparatus has a means (e.g. control unit 2, Crampton Fig. 1) to control the position of the 
aforementioned second three dimensional measuring means (e.g. an SL scanning means - Crampton 
column 6, lines 8-22). The arm (e.g. arm 1 - Crampton Fig. 1 and coliunn 2, lines 34-47), and consequently 
the 3D measuring means, of Crampton' s apparatus moves according to a "library" of predefined range 
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images (Crampton column 17, lines 54-67 to column 18, lines 1-6). According to Crampton (Crampton 
column 17, lines 54-67 to column 18, lines 1-6 and column 26, lines 51-55), this library may include rough 
default range images (e.g. cylinders or cubes). Thus, Crampton demonstrates controlling the position of the 
second three dimensional shape measuring means, said control means being controlled based on said rough 
shape information. 

88. The teachings of Crampton are combinable with those of Niem because they are analogous art. 
Specifically, both Crampton and Niem disclose methods and systems for 3D reconstruction using SL 
methodologies. Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
apphcant's claimed invention, to control the position of the 3D measuring means (e.g. the camera of Niem 
or the aforementioned 3D scanning means of Crampton) in accordance with a rough depth image, as taught 
by Crampton. Recall that, in the method of Niem, a rough 3D reconstruction (SFS reconstruction) is 
derived prior to deriving a detailed 3D reconstruction (SL reconstruction) of the object. Given this, it would 
have also been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to 
control the position of the 3D measuring means (e.g. the camera of Niem or the aforementioned 3D 
scanning means of Crampton) in accordance with the previously derived 3D SFS reconstruction (e.g. by 
simply adding it to Crampton's Ubrary of range images). The motivation for doing so would have been to 
provide an automated means to capture images and reconstruct a 3D model from multiple viewpoints about 
the object. The motivation to use the SFS reconstruction as the rough shape guiding the position control 
would have been to refine the SFS reconstruction by accounting for concavities. Combining the teachings 
of Crampton and Niem, in this marmer, would yield a modeling apparatus that conforms to the 3D 
modeling apparatus set forth in claim 67. 

89. The following is in regard to Claim 53. Claim 53 recites substantially the same limitations as 
claim 67. {Moving is essentially the same as controlling the position and the three dimensional shape input 
means having higher three dimensional input accuracy and the three dimensional shape input means having 
lower three dimensional input accuracy are essentially the same as the aforementioned detailed and rough 
shapes, respectively). Therefore, with regard to claims 53 remarks analogous to those presented above with 
regard to claims 67 are apphcable. 
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90. Claims 50-51 are rejected under 35 U.S.C. 103(a) as being unpatentable over Niem, in view of 
Batlle et al., in fiirther view of Crampton. 

91. The following is in regard to Claim 50. As shown above, the teachings of Niem and Batlle et al., 
when combined in manner discussed above, adequately satisfy the limitations of claim 1. Neither Niem nor 
Batlle et al. expressly show or suggest a 3D modeling apparatus that fiirther includes a moving means for 
moving said projection means and said image input means. 

92. Crampton discloses an SL scanning apparatus for obtaining 3D models of scanned objects. The 
scanning apparatus has a means (e.g. control unit 2, Crampton Fig. 1) for moving projection means (e.g. an 
SL scanning means - Crampton colimm 6, lines 8-22). See Crampton column 2, lines 34-47. 

93. The teachings of Crampton are combinable with those of Niem because they are analogous art. 
Specifically, Crampton, Batlle et al. and Niem all disclose methods and systems for 3D reconstruction 
using SL methodologies. Therefore, it would have been obvious to one of ordinary skill in the art, at the 
time of the applicant's claimed invention, to change the position of the SL projection means in the 
apparatus, implementing the extended SFS method discussed above, using an arm-like apparatus such as 
that which is depicted in Fig. 1 of Crampton. The motivation to do so would have been to provide an 
automated means to capture images and reconstruct a 3D model fi-om multiple viewpoints about the object. 
Combining the teachings of Niem, Batlle et al., and Crampton, in this manner, yields an 3D modeling 
apparatus that conforms to that which is set forth in claim 50. 

94. The following is in regard to Claim 51. As shown above, the teachings of Niem, Batlle et al. and 
Crampton, when combined in manner discussed above, adequately satisfy the limitations of claim 50. In a 
maimer analogous to that which was presented above with regard to claim 67, the modeling apparatus 
obtained by combining the teachings of Niem, Batlle et al., and Crampton can be straightforwardly 
modified by one of ordinary skill in the art so that the moving means moves the projection means 
(analogous to the second 3D shape measuring means discussed above relative to claim 67) and image input 
via this projection means (i.e. its movement) is based on said rough shape. See the discussion above 
relating to claim 67. Therefore, combining the teachings of Niem, Batlle et al., and Crampton, in this 
maimer, yields an 3D modeling apparatus that conforms to that which is set forth in claim 51. 
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Citation of Relevant Prior Art 

95. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: 
[1] The Visual Concept for Silhouette-Based Image Understanding. Laurentini. IEEE, PAMI, 
1994. 

Laurentini introduces the concept of visual hulls as a shape-from-silhouette method for 
generating three dimensional models of objects from 2Dimages. The paper provides more 
theoretical background relating to the concept of visual hulls than does Wojciech et al. 
[2] Construction of 3D Models of Objects Using Combination of Shape From Silhouette and 
Shape from Structure Light. Tosovic. July 2001. 

Tosovic propose building a rough reconstruction of an object using shape-from-silhouette 
methods and refining this initial model by using structured hght techniques. 

[3] On Combining Shape from Silhouette and Shape from Structured Light Tosovic, et al. 2002. 
Tosovic et al. propose building a rough reconstruction of an object using shape-from-silhouette 
methods and refining this initial model by using structured light techniques. 

[4] Solid Model Acquisition from Range Imagery. Reed. 1998. 

Reed suggests (section 2.3. 1) extending a visual-hull based reconstruction method by utiUzing 
range data or range imagery. Structured hght techniques are typically classified as active 
ranging techniques. 

[5] Stripe Boundary Codes for Real-Time Structured-Light Range Scanning of Moving Objects. 
Hall-Holt, et al. IEEE ICCV 2001. 

Hall-Holt et al. use structured light range scanning to capture 3D models of moving objects. 
Hall-Holt et al. project striped binary patterns that vary temporally onto the scanned objects. 
Of particular relevance to the Apphcant's invention is Hall-Holt et al.'s discussion of stripe 
boundary codes. Hall-Holt et al. assign codes to stripes of the illumination pattern and 
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observes code transitions at the stripe boundaries (i.e. pattern border regions). As in the 
Applicant's claimed invention these are used to resolve ambiguities that may arise in deriving 
the correspondences between images (in this case the projected image and captured image). 
The graph shown in Hall-Holt provides a decision framework for resolving such ambiguities. 
It is similar to the correspondences shown on page 45 of the Applicant's disclosure. 
[6] Range Data Acquisition Using Color Structured Lighting and Stereo Vision. Chen, et al. 1997. 
Chen et al. demonstrate the combination of color structured hghting with stereo vision 
techniques. In particular, a color pattern is projected onto an object and stereo images are 
captured. Correspondences are derived between the stereo images as opposed to between a 
captured image and the projected image. 
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